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ABSTRACT 



The Atlanta Public School System conducted four major 
programs during the summer of 1971, These were Youth-Tutor ing-Youth , 
Community Schools, Handicapped Children, and Prekindergarten 
Instruction, Evaluation of the Prekindergarten (Pre-K) project was 
contracted to the Laboratory for Applied Behavior Research at Georgia 
State University. The Laboratory agreed to: (1) provide consultants 

to assist in the implementation of behavior modification techniques 
for use with two basic reading programs utilized in the Pre-K 
classes, (2) evaluate the use of each reading program in the Pre-K 
classrooms, irrespective of the program 1 s association with behavior 
modification procedures, (3) describe the overall academic activities 
of the Pre-K classes, and (4) suggest specific changes for future 
Pre-K projects. The BRL Skills and Concepts Survey was used as the 
evaluation instrument for this program. After the pre- program 
measures were taken, the children were divided into two groups; (1) 
those whom the teacher considered emotionally, be ha vi orally , and 
intellectually ready for the BRL reading program, and (2) those whom 
she considered too young or otherwise not adequately prepared for the 
BRL format, A main result of this study was the finding that the BRL 
reading program is superior to others. (Author/CK) 



O 

ERIC 



OFFICE OF EDUCATION 
THIS DOCUMENT HAS BEEN REPRO- 
DUCED EXACTLY AS RECEIVED FROM 
THE PERSON OR ORGANIZATION OHIO 
INATING IT. POINTS OF VIEW OR OP!N= 
IONS STATED DO NOT NECESSARILY 
REPRESENT OFFICIAL OFFICE OF EDU- 
CATION POSITION OR POLICY. 



RESEARCH AND DEVELOPMENT REPORT 



Vol . V, No. 3 



Summer , 1971 



PREKINDERGARTEN PROGRAM EVALUATION 
SUMMER^ 1971 

Funded Under 15EA Title I, P. L. 89-10 



Prepared by 

Dr, Teodor o Ay lion, Director 
The Laboratory for Applied Behavior Research 
Georgia State University 

Assisted by 

Shawn Michael Boles, Ph,D, 

George O’Neill, B.A* 

Kathleen Kelly, M. A. 



Dr, Jarvis Barnes 
Assistant Superintendent 
for Research and Development 



Dr* John W, Letson 
Superintendent 



Atlanta Public Schools 
224 Central Avenue, S, W, 
Atlanta, Georgia 




1 



ACKNOWLEDGMENTS 



The authors wish to express appreciation to all the teachers 
involved in the Summer Prekindergarten Program, especially to Ms, 
Helen Fitzgerald and Ms. Sarah Singley, both of whom worked closely 
with the consultants * 



Special thanks are expressed to Ms* Betty Fromhagen for her 
voluntary help with the REBUS program at E. S. Cook Elementary 



School. 



Hereafter fictitious names have been used for classroom 



teachers . 



O 

ERIC 



TABLE OF CONTENTS 



INTRODUCTION 



1 



THE BRL— SULLIVAN PROGRAM 



Description of the BRL Program 4 

BRL Program With Behavior Modification . 4 

Subjects ..*••,....,.**..****«*• 4 

S e 1 1 i n g •«»#■•*•■*••••••»■•'*•■***•** 4 

Personnel * . • * 5 

Program Evaluation . * * * « 5 

Procedures 5 

BRL Reading Group . * .. * . * * - - 6 

Non— BRL Group • » 6 

Other Behavioral Interventions . S 



BRL- SULLIVAN PROGRAM WITHOUT BEHAVIOR MODIFICATION 
M . A. Jones . . * 

JjUCtciG 

Slater . , , . 

Williams 



The Results of Using the BRL-Sullivan Program 

in Prekindergarten Classes * 

Within School Analyses ... 

Between School Analyses . » 

The BRL-Sullivan Program in Prekindergarten .. ...... 

Re commends fc ions for BRL 

THE PEABODY' REBUS 



Description of the Rebus Program . . . * 



The RebtiB Program With Behavior Modification . 



The Peabody— P.eb us Without Behavior Modification ,.*•*••••# 

Harris * 

Fitts ■*«(*••>**••>••• ************* 
Ware * * * 

The Results of Using the Peabody-Rebus Program 

in Prekindergarten Classes * 



9 

11 

12 

13 



14 

13 

16 

17 

IB 



19 

20 

26 

26 

27 

28 

30 







3 



TABLE OF CONTENTS (Centd . ) 



The Rebus in Prekindergarten 

Summary and Conclusions **.*,*• 
APPENDIX 

Tables * . . 

Test Instrument Used With BRL Program 



Page 



32 

34 

A-l 
A— 6 



Number 

1 

2 

3 

4 

5 



LIST OF TABLES 

Pa ge 

WITHIN SCHOOL CHANGE FROM PRETEST TO POSTTEST FOR 

THOSE SCHOOLS ASSOCIATED WITH THE BRL PROGRAM A-l 

RESULTS OF THE TEACHER QUESTIONNAIRE DEALING 

WITH THE BRL PROGRAM A-2 

REINFORCEMENTS A-3 

PER CENT OF STUDENTS EXPOSED TO EACH SECTION 

OF REBUS WORKBOOK I A-4 

RESULTS OF THE TEACHER QUESTIONNAIRE DEALING 

WITH THE REBUS PROGRAM A-5 




-ii- 



4 



Prekindergarten Project Evaluation 
Summer* 1971 



INTRODUCTION 

The Atlanta Public School System* operating with monies received from 
the Federal Government (Public Law 89 — 10* Title I)* conducted four major 
programs during the summer of 1971* Those programs were Youth— Tutoring- 
Youth* Community Schools* Handicapped Children* and Prekindergarten 
Instruction.^ Evaluation of the Prekindergarten (Pre— K) project was 
contracted to the Laboratory for Applied Behavior Research at Georgia 
State University a The Laboratory agreed to: 

1* Provide consultants to assist in the implementation of behavior 
modification techniques for use with two basic reading programs 
utilised in the Pre— R classes. 

2* Evaluate the use of each reading program in the Pre-K classrooms* 

irrespective of the program 1 s association with behavior modification 
procedures. 

3. Describe the overall academic activities of the Pre-K classes* 

4- Suggest specific changes for future Pre— K projects. 



A description of each of the four programs may be found in "Application 
for Federal Assistance for the Education of Children from Low— Income Families 
for Fiscal Year 1971* rf Atlanta Board of Education* April 30* 1971<* 
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Pre-K classrooms were established at nine Title I elementary schools 
in the Atlanta area. These schools were Cook, Grant Park Primary, Harris, 

M.A. Jones, Luckie, Pitts, Slater, Ware, and Williams. All of the schools 
were chosen because they were located in lower socio-economic areas. In 
addition, two of the schools were associated with large public housing 
projects— Pitts with Perry Homes and Williams with Bowen Homes. Projected 
enrollment was thirty children at each of the nine schools giving a total of 
two hundred and seventy children in the Pre— K program. 

Program activities for the children began on June 21, 1971 and ended 
July 30, 1971. The children spent a total of six weeks (twenty-nine days) 
in the program. The teachers, in addition to the six weeks spent working 
with the children, underwent one week pre-service and post-service sessions. 

A major objective of the Pre— K program, as stated in the original 
proposal, ^ was "To provide a structured readiness program designed to develop 
the pre-reading skills of Pre-Kindergarten pupils, and first grade pupils 
who have not had kindergarten experience. " In order to facilitate the 
attainment of this objective, two fully developed programs for the teaching 
of reading were selected for use in the Pre— K classrooms. 

The first of these programs was Behavior Research Laboratory 1 s Sullivan 
Reading Program (BRL) . The BRL program was used by classes at Grant Park 
Primary, H. A. Jones, Luckie, and Slater schools. Teachers at M. A. Jones, 
Luckie, and Slater simply followed the workbook for the BRL program, while 
the teachers at Grant Park Primary used the BRL materials in conjunction 
with specific behavioral techniques designed to increase the efficiency 
of teaching Pre-K children. Ms. Kathleen Kelly was the consultant to 
Grant Park Primary. 

The second program selected for use in Pre-K classes was the Peabody 
Rebus Reading program. The Rebus was used at Cook, Harris, Pitts, and 
Ware schools. As with the BRL program, teachers at three of the Rebus 
schools (Harris, Pittd,and Ware) simply integrated the Rebus program into 
their regular classroom activities, while the teacher at Cook used the program 
in conjunction with specific behavioral techniques, Mr. George O'Neill 
was the consultant to Cook school. 



3 Ibid 
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Teachers at Williams school used neither the BRL nor the Rebus 
materials to teach pre-reading skills to the children. Rather they 
relied on what was termed an "eclectic 11 approach. This approach con- 
centrated on exposing the children to a large number of concepts (e,g, 
colors j sizes j space relations) in a variety of different situations 
throughout the course of the Pre-K program. Pupils at Williams school 
were used as controls for the BRL program. 

The following two sections describe the use of the BRL and Rebus 
programs in the Pre-K classes. Within each section the same format will 
be followed. Firsts the program will be fully described. Second,, the use 
of that program in the behavior modification school will be explained. 
Thirds the program 1 s use in the other schools will be described. Fourth^ 
a comparison of the pro gram T s use with and without behavior modification 
will be given. 

The last section of the report deals with other academic activities 
used in the Pre-K classes. Differences in teacher approach., the setting 
and meeting of objectives^ and suggestions for change will be discussed. 



THE BRL— SULLIVAN PROGRAM 



Description of the BRL Program 

The Sullivan reading program is a product of the Behavior Research 
Laboratories in Palo Alto, California* This programmed reading series is 
designed so that after initial teacher-directed phases, the child may proceed 
to work at his own pace. The authors of this series have attempted to arrange 
reading material so that the user is shaped into making responses at a very high 
rate of accuracy. Motivated by this success, the child 1 s interest is maintained 
and new components of reading, at increasing levels of complexity, are gradually 
introduced. The Sullivan program, then, is designed to produce a condition of 
11 errorless learning 11 in reading* 

"Readiness in Language Arts 11 is the first stage in the Sullivan reading 
series and was the only level used in this program* The format consists of 
an imaginary trip taken by several animals where shapes, letters, and colors 
are introduced. As the animals in the book make these discoveries, the 
children are learning, too, and, by the end of the book, can read a limited 
number of letters. This level of the reading series is teacher— directed and 
the children 1 s responses are oral. Materials used include the large, black- 
board size book with an easel stand and the teacher T s manual* Each step is 
clearly directed with page by page instructions for the teacher* 



BRL Program With Behavior Modification 

Subjects * Twenty— two children originally registered for the Summer 
Pre— K Program. Fourteen children completed at least 15 days of the program. 
The age of those children completing the program ranged from 4 to 6 years 
for an average of 4*8 years of age. The remaining eight children attended 
from one to six days before withdrawing from the program* 

Setting. One of the regular kindergarten classrooms In the Grant 
Park Primary School was used for this study. Equipment in the room consisted 
of 4 child— size tables with chairs, a listening station with headphones 
for 6 children, a play kitchen, a block-building area, and an artist f s easel 
f or painting. 



Additional facilities in the school consisted of a large play area for 
rainy day activities * the school playground* and a cafeteria eating area* 

Personnel, One regular kindergarten teacher and one aide were present 
du mg the 6— week program. Four Atlanta Area Teachers Educational Service 
(AATES) teachers were in the classroom for two weeks each during the study. 

In addition* the school nurse assisted the teacher and served as an additional 
aide throughout the course of the program, 

Mrs, Cunningham* the classroom teacher* took a personal interest in each 
child. The aide and the school nurse shared her enthusiasm and the three 
together conducted a well— coordinated program* Having two additional adults 
in the classroom (each team of AATES teachers for 2 weeks) required special 
arrangements* The first pair of AATES teachers worked very well in the 
class j the second pair* however* often arrived after 9:00 a*m. The exact 
function of the AATES teachers was not made clear- part of the time they served 
as participants* but at other times they were simply observers. 

Program Evaluation* The I3RL Skills and Concepts Survey was used as 
the evaluation instrument for this program. All 12 children present during the 
first week of the program were given the survey checklist* Post— program 
measures were taken for those same children only* Thus* any children not 
entering during the first week of the program were not considered in the 
evaluation* Survey checklists were administered and evaluated by program 
consultants* The teachers had no information regarding items on the survey 
until after the program had been completed. 

Procedures * After the pre— program measures were taken* the children 
were divided into 2 groups: 1) those whom the teacher considered emotionally* 

behavioraXXy* and intellectually ready for the BRL reading program* and 2) 
those whom she considered too young or otherwise not adequately prepared for 
the BRL format. Each of these groups followed the regular class schedule 
in all activities except for a 15 minute period each day when their particular 
intervention took place. 



BRL Reading Group , Directed by Mrs, Cunningham* this group consisted 
of seven children whose average age was 5*3 years. The standard BRL procedur 
were followed as directed by the teacher T s manual. The teacher used primary 
reinforcement in the form of pre-sweetened cereal on an intermittent basis. 
Thus* she gave this form of reinforcement not after each correct response* 
but rather after the child had made several correct responses. This schedule 
was followed in order that the child not become dependent upon the primary 
reinforcer for every response. Further* intermittent delivery of the rein- 
foroement results in much longer responding when the primary reinforcement 
is faded out. Social reinforcement* however* was delivered almost continu- 
ously for correct answers ; this included smiles* hugs* and praise by the 
teacher. 

In an effort to facilitate the left-right discrimination* each chi! d 
in this group wore a red ribbon on his left wrist* the same color as is used 
in the large book for !T left side of the page. 11 The seven children were 
divided into 2 groups of 3 to 4 children each* Both groups met regularly 
each day for the fifteen minute period. While one group was engaging in 
the BRL activity* the other group would be with one of the aides in another 
area of the room. At the end of the session* the two groups would switch 
places and the second group would participate in the BRL program, 

Non-BRL Group . Six children who were considered unprepared for the 
formal BRL reading program comprised the second group in this study. The 
average age was 4*2 years. An aide* assisted by the program consultant* 
conducted this group for 15 minutes each day. In an attempt to evaluate 
the BRL program from a different direction* the following format was 
designed. The stated objectives of the BRL program* on the level used 
by the first group* were outlined into four categories as discussed below. 

The question was asked; Could these objectives be taught directly to the 
children* by-passing the BRL format? In order to evaluate these procedures* 
each child was exposed to ail four categories daily- Five presentations 
were made from each category so that the child had 20 possible correct 
responses each day. The four major groups of BRL objectives were as 
follows; 

1, Concepts: Left, right, top, middle, bottom, (Aide: "Which hand 
is this?” "Where on the card is this shape, "etc.) 

2, Numbers: 1-6 (Aide used 3'* x 5" cards with one to six objects 
printed on card. Aide- "How many ^objects] do you see?") 



3. Color: Red, green, blue, yellow, black, white, green 3 orange, 

' brown, (Aide: "What color is this?") 

4 Shapes: Triangle, square, circle . (Aide used cardboard shapes. 

Aide: "What shape is this? 11 ) 

A daily record was kept of each child f s responses to the 20 stimulus 
items presented by the aide* Additional items had to be included in the 
concept and colors categories on Day 2 of the study. The children were 
responding at a level of 60 % to 75 % accuracy and it was decided that the 
stimulus items were too simple. 

The original design for this non-BKL intervention was the multiple 
baseline technique. After stable measures had been gotten on all four 
behaviors, one of the four categories would utilise reinforcement for 
correct responses. After this category had demonstrated the effect of 
reinforcement on correct rate of responding, a second behavior would also 
be reinforced. The two remaining behaviors would remain a baseline state 
until successively reinforced. It was found, however, that the repeated 
presentations of the stimuli alone were sufficient to increase performance 
rate. It is possible that systematically organizing the material into 
concise form and then regularly exposing the children to the material can 
effect performance increases not found in less direct presentations. 

The revised intervention, then, was to systematically present the material 
with no extrinsic reinforcement. 

If a child answered correctly, the aide answered, "That^ correct, 
(name of child). 11 If the child answered incorrectly, the aide responded, 

"No (name of child), this color is red," or whatever the correct response 
should have been. 

One child in this group, David, began to display tantrums and balking 
behaviors during session 3 of the intervention. It was decided to utilize 
intermittent primary reinforcement for him only. Beginning with his next 
session and until the completion of the study, David received primary 
reinforcement and praise for his correct responses. He worked apart from 
the group beginning with Session 3- 



Other Behavioral Interventions . In addition to the efforts directed 
specifically toward pre-reading skills* several other suggestions made by 
the behavioral consultant were carried out by the teacher. A standard kitchen 
timer was used to facilitate movement from one activity to another. Before 
each activity was completed* the teacher and aides would instruct the chil- 
dren where to go next. If they went promptly to the assigned activity 
area and were settled down before the timer rang* she would intermittently 
deliver a small amount of primary reinforcement (sweetened cereal). She 
would also assign the status roles (first in line* message carrier* etc*) 
to those children who followed her directions most quickly in various activi- 
ties. This included participation* cooperating with another child* paying 
attention* and so on. 

The results from this classroom* then* are the interaction of two 
important variables: l) the BRL reading program and 2) behavior modification 
techniques , The data will be effected by both of these agents of change 
and the effects of each cannot be separated within the scope of this study. 

One alternative to this analysis problem would be to compare the results 
of the BRL-Behavior Modification classroom with other BRL classrooms not 
using behavioral techniques. What follows is a description of these classes. 



BRL— Sullivan Program Without Behavior Modification 



In addition to its use at Grant Park Primary the BRL program was 
also used at' Jones, Luckie, and Slater schools. Teachers at these 
schools were not given any specific instructions as to how to run the 
program in their individual classrooms. As a consequence, the use of the 
BRL materials varied greatly from class to class. Rather than attempt 
to describe the classes as an aggregate, each teacher's use of the BRL 
will be dealt with separately. In each case, the description of procedures 
is based upon random visits paid to the schools by an evaluator from the 
Laboratory for Applied Behavior Research. Evaluation visits were randomly 
scheduled and the evaluator arrived unannounced at each location. Usually 
about three hours were spent at each school during each visit. In addition 
to ev al uation visits, each BRL Pre-K classroom was pre-tested and post- 
tested using a questionnaire based upon the BRL— Sullivan Skills and Con- 
cepts Survey kit. (The nature of this test and the results associated with 
its use will be described fully in a later section.) 

M. A. Jones . The Pre-K class at this school was under the direction 
of Mrs. S mi th.. She was assisted throughout the six weeks by a variety of 
personnel including AATES interns, aides, parents, and NYC workers* The 
total enrollment at M. A, Jones was thirty-one children. Of these thirty- 
one, twenty-seven attended more than fifteen days. 

The first evaluation visit was paid to M. A. Jones on July 6, 1971. 

Due to the low attendance following the fourth of July weekend (N=14 of 
31 ) and the fact that the children were undergoing eye tests, no BRL 
activities were conducted on this date. Mrs. Smith reported that the 
BRL program was normally conducted from 10:00 a.m. to 10:50 a.m. each day. 
Several small groups of children (from 5-7 children in each group) were 
rotated through a variety of activities during this period. These acti- 
vities were the BRL materials, lotto games, pegboards, and art. The 
average amount of time spent in a particular activity was about 15 minutes 
according to Mrs. Smith, 

The second evaluation visit was made on July 15, 1971- Three 
groups of children (N=6, 7, and 8) were exposed to the BRL materials. 

Each group spent 15 minutes at the BRL activity, which was conducted by 
one of the aides. The following facts were noted about the teaching: 



(1) The aide used "brown" and "tan" as synonyms in talking about colors, 

(2) She did not correct children who confused the left and right sides 
of the BRL book, (3) She did not stop fighting which occurred during 
the lesson. (4) Both the aides and the children 1 s concentration on 
the material was impaired by the continuous annoyance caused by gnats 
in the classroom. 

The third evaluation visit was made on July 23* 1971 and lasted one 
hour, Mrs, Smith used the BRL materials for 25 minutes with all the 
children in a single group (N— 22). Mrs. Smith had not read the BRL 
materials prior to conducting the lesson. The materials had been assigned 
to one of the AAT3 interns who did not arrive on this day. Mrs, Smith 
spent most of the time having the children respond to questions associated 
with such concepts as left— right, firs t — last , middle, more than, squares, 
circles, and color mincing. Two major disadvantages were noted in the 
use of the BRL program with a single large group; 1) Many of the children 
on the periphery of the group did not become involved with the material. 

(2) Due to the large sise of the group it was difficult for Mrs. 
to correct those children who responded incorrectly to her questions. 

Use of the BRL materials at M. A. Jones appeared to be somewhat 
flexible. The specific ins true tio nal techniques used (small groups, 
large groups, etc,) fluctuated as a function of attendance, outside 
activities, and the teacher r s perception of how well the children were 
progressing, Mrs, Smith seemed to have some anxiety about the pre and post 
testing of the children on the BRL material. Although assured by the 
evaluator that the tests were to be used only to evaluate the BRL program 
itself, she persisted in wanting to know the nature of specific test items. 
Upon receiving information from the evaluator that the test was generally 
concerned with animal names, colors and such, she proceeded to spend time 
teaching just those items to the children. This probably did not inflate 
the pupils V performance scores, since Mrs. Smith was never given a copy 
of the test until the end of the Prc-K program. Still it indicates some 
of the difficulties one encounters in trying to obtain reliable information 
about programs used in a public school setting. 




Luckie* The Pre-K class at this school was under the direction 



of Mrs. Carnes and her co-teacher Mrs# Bell# In addition to these two 
teachers a health aide was normally present. Volunteer workers also helped 
the two teachers in several of the tasks associated with conducting the 
class (e*g. story telling* preparing snacks* assisting on field trips). 
During the first weeks of the Pre-K program at Luckie a teen-age boy 
worked with the children. He provided a role model for the boys to follow. 
He also helped the teachers by shepherding the boys to wash their hands* etc. 

It is difficult to assess the actual attendance figures for the Luckie 
Pre-K class. Mrs* Carnes turned in two separate attendance forms to the 
Laboratory for Applied Behavior Research# These forms conflicted with one 
another with respect to how many children were actually enrolled in the 
program. A conservative estimate would be that twenty-seven children 
entered the program. Twenty— one of them completed more than 15 days in 
the pro gram . 

The first evaluation visit to Luckie was made on July 1* 1971* 

On this date there were 19 children present* All of the children were 
exposed to the BRL materials* either by Mrs* Carnes or by Mrs* Bell, 

The procedure was for the children to be divided up into several small 
groups (>N=23 in each group*) Each group was then exposed to the BRL 
program for a short time* The amount of time spent on the BRL material 
appeared to be a function of the individual child T s attention span (i,e* 
children were allowed to leave their BRL group if they became bored or 
restless* These children engaged in other activities such as free play 
or art)* An average session seemed to run about 10 minutes. Mrs* Carnes 
and Mrs, Bell reported that they were using the BRL program every day with 
the children. They were rotating the children in such a fashion that each 
teacher saw each child over the course of several days instruction. 

The second evaluation visit was made on July 12* 1971* Fourteen 
children were present. Mrs# Carnes and Mrs, Bell had developed a mimeo — 
graphed form for keeping each child r s BRL progress. This form* which 
provided a record of the number of pages a child had covered in the BRL 
book* was used in building the small groups for each BRL lesson. In 
addition to the use of a record form* Mrs* Carnes and Mrs. Bell had de- 
cided to permanently divide the children for the purposes of BRL instruc- 
tion* Mrs, Carnes always worked with the same groups of children* as 
O [ Mrs. Bell. This procedure allowed the two teachers to give more 
ERIC ■sonel attention to their respective children, 

1S1- 



The third evaluation visit was made on July 23* 1971 and lasted one 

hour. Nine children were present on this date, Mrs. Carnes and Mrs* 

Bell continued to use those procedures which had been instituted by the 
time of the second evaluation visit. They seemed to have some difficulty 
in teaching the children to reliably discriminate red from orange in the 
BRL book. They were forced to rely on objects in the room 5 rather than 
the book, to teach the differences. This difficulty could be resolved 
by using a more saturated red in printing the BRL book. 

Mrs, Carnes and Mrs. Bell appeared to work very well together through- 
out the Pre— K program. Their approach to the use of the BRL materials 
was rather innovative. The major drawback associated with their use of 
the BRL program seemed to be not so much with the teachers as with the 
population of students with whom they were dealing* The children at 
Luckie seemed, to the evaluator, to appear and disappear from the class- 
room with astonishing rapidity. This fluctuation in attendance made the 
use of the BRL materials rather difficult. 

Slater. The Pre— K class at this school was conducted by Mrs. Evans, 
assistance in running the class was provided by AATES interns* aides* and 
NYC workers. The total enrollment at Slater was thirty— two children* 
Twenty— eight children attended for more than fifteen days. Due to time 
constraints only two evaluation visits were made to Slater. 

The first evaluation visit was made to Slater on July 8, 1971* 
Twenty-five children were present. Mrs. Evans had divided her children 
up into three rather large groups (N— 10— 11. in each group). These groups 
were exposed separately to the BRL material for approximately 15 minutes 
each day. On the day of the evaluation visits however, all the bhildren 
were given their BRL lesson together (Mrs. Evans had told the evaluator 
previously that she thought it possible to use the BRL with large groups.) 

The second evaluation visit was made on July 19* 1971- Twenty-three 
children were present. By this date Mrs. Evans was using the BRL materials 
with the total group, only bo review those parts of the program that the 
children had been exposed to in smaller groups. Mrs. Evans reported that 
her major emphasis in using the BRL program was to work with those 
children who would definitely be attending Slater in the fall of 1971* 
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These returning children (N-=ll) were placed together in one of the three 
groups * and consistently exposed to the BRL program. The other two groups 
of children were treated in a somewhat more cursory fashion in terms of 
their contact with the program (i.e, the evaluator got the sub j ective 
impression that there was less concern with the children 1 s understanding 
of the BRL program in the two groups of non-returning students* than in 
the group of returning students), 

Williams,. The Pre- K class conducted at Williams did not use the BRL 
program. Instead* the children at this school were exposed to a variety 
of educational experiences designed to teach the alphabet* color discri- 
mination* etc, These children were used to provide a control condition 
for the schools in which the BRL program was used* 

The Pre-K program at Williams was under the direction of Miss Timms* 
Miss Timms was absent for 10 days during the six weeks. The class was 
conducted by Mrs. Owen in her absence. Miss Timms and Mrs, Owen were 
assisted by AATES interns* aides* volunteers* and NYC workers in running 
the class. Thirty- four children were enrolled in the Pre-K program at 
Williams, Of these thirty- four* twenty-three attended for more than 15 
days . 

The first evaluation visit to Williams was made on July 17* 1971. 
Twenty-two children were present. The evaluator f s main impression was that 
the classroom had an intolerably high noise level. The noise stemmed 
primarily from the fact that no control seemed to be exercised over 
children who banged on the piano* screamed* banged blocks together* etc, 

A variety of activities (e.g, listening station* art* blocks* cooking) 
were continuously available to the children. Each child could choose 
to participate in as many of the activities as he wished. The length 
of participation was determined by the child himself. Children often 
moved from activities such as the listening station after as little 
as five minutes. This short exposure time* coupled with the noise level 
in the room* seemed to preclude any sort of lasting academic learning 
experiences. 
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The second visit to Williams was made on July 20, 1971, Twenty-one 
c h:Ll-d.ren were present. The noise level in the class had been reduced somewhat 
through rearrangement of the room's furnishings. This new arrangement 
served to divide the room into specific areas for the activities of cook- 
ing and baking, housekeeping, art, television and listening station, and 
a play area with a reading corner. The noise reduction achieved was from 
the new physical arrangement not from any controls instituted over the 
childrens' behavior. The children still were allowed to choose both the 
activities they engaged in and the length of time they participated. 



The Results of Using the BRL-^Suliivan Program in Prekindergarten Classes 

In order to assess any benefits associated with the use of the BRL 
program in Pre— K classes, pre and post tests were administered to the 
children at Grant Park Primary, Jones, Luckie, Slater, and Williams, 

The test used was developed by the staff of the Laboratory for Applied 
Behavior Research and is based upon the Skills and Concept Survey kit 
marketed by Behavior Research Laboratory. The test instrument is appended 
to this report. Use of the test involves asking an individual child 
questions about items on two stimulus cards. The questions are designed 
to determine whether the child: can name 8 colors; can discriminate left 
from right j can count up to four; can distinguish letters from pictures 
and numbers; can name animals and letters; and can follow simple instruc- 
tions using the concepts of first, middle, top, and bottom. 

Pre— test at each of the four BRL schools and at Williams was done 
during the last week of the program, from July 26 to July 29, 1971, 

Care was taken to insure that the pre and post test situations were as 
similar to one another as possible. Thus, each child was tested in the 
same place at his school on each of the two test occasions. Additio nall y, 
the testers made sure that as many children as possible had the same 
tester for the two test administrations, (it was not always possible to 
hold to this latter constraint. However, approximately 80 % of the 
children tested did have the same tester on both testing occasions). 

One hundred and forty-si^c children were enrolled in the five schools 
associated with the BRL program. Of these children, one hundred and thirteen 
were present at least fifteen days. Complete pre and post test results 
were obtained for seventy-five children (N=9 at Grant Park, 19 at Jones, 
at Luckie, 19 at Slater, and 15 at Williams). 




The discrepancy between the number of children enrolled and the number 
of children tested was due to two major factors. First, several children 
enrolled after the initial week of the program and thus were not present 
for pre-testing. Secondly, vacations, withdrawals, and absenteeism 
reduced the number of children present at the time the post=te$t was 
administered . 

A raw score for each of the seventy— five children was obtained by 
subtracting the number of correct answers the child gave on the pre-test 
from the number of correct answers he gave on the post-test. (No child 
made a perfect score of 37 correct on the pre-test. Thus the raw score 
for each child could have been either positive or negative.) The raw 
scores indicate the difference between each child 1 s performance on the 
two testing occasions. The raw scores are from an ordinal scale of un- 
known distribution and therefore are appropriately analyzed by nonpara— 
metric statistical procedures. Two separate analyses of the data were 
conducted. The first of these dealt with the extent to which the subjects 
within each of the five schools had changed from pre-test to post-test. 

The second set of analyses concerned comparisons between the three 
separate types of schools - BRL with behavior modification, BRL without 
behavior modification, and the control school* 

Within School Analyses . The hypothesis with each of the 

schools was that the children would get more it correct on the post— test 
than the pre-test. This was a one-tailed hypothesis and was tested against 
the Null hypothesis that the children within each school would show no 
change from pre-test to post-test, or that the change would be negative 
(i.e* more correct items on the pre— test than on the post— test) . 

The Wilcoxon Signed Ranks test^ was employed In testing the two 
hypotheses against one another for each school. The results of th6s e 
analyses are reported in Table i. Inspection of this Table reveals 
that the children in Grant Park primary, where the BRL program was used 
in conjunction with behavior modification techniques, did significantly 
better on the post-test than they had on the pre-tent. Children at 



^Sidney Siegel, Nonparametric Statistic : - for the Behavioral Sciences, 
McGraw-Hill: New York, 1956. pp. 75—83- 
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Luckie 5 Jones $ and Slater schools (where the BRL program without behavior 
modification was employed) ^ also improved significantly from pretest to 
posttest. 

The only school that did not show a significant increase across the two 
testing sessions was Williams. It will be recalled that the children at 
Williams were not exposed to any systematic reading program^ including the 
BRL. 

Between School Analyses. The within schools comparisons establish that 
the BRL program^ both with and without behavior modification^ leads to a 
significant increase in test scores* The question now arises as to whether 
the increase is greater in Grant Park^ where behavior modification was used., 
than in the three schools that used the BRL program without behavior 
modification* In order to answer this question the scores for the students 
at Jones (N^LQ)* Luckie (N= 13)* and Slater (N -19) ^ schools were combined 
to form a class of children who had been exposed to the BRL program without 
behavior modification (BRL alone group). Scores for the Grant Park children 
were then compared with those of the BRL alone group to test the hypothesis 
that the children at Grant Park had increased their test scores more than had 
the children in the BRL alone group. 

4 

A sing le“ta lied Mami-Whitney U test was used to compare the children 
at Grant Park to the children in the BRL alone group. The scores for the 
nine children at Grant Park and 51 children in the BRL alone group were 
ranked together. From the rankings a B value of 142 was calculated* This 
is equivalent to a a value of 1.81 which is significant at the ,03 level 
(one tailed)* This indicates that the BRL Program with behavior modification 
increased test scores more than did the BRL without behavior modification. 

(The alternative interpretation that the BRL alone group was initially different 
from the Grant Park group may safely be eliminated^ since a comparison of the 
pretest scores for the two groups showed no significant difference between 
them, ) 



4 lbid, pp. 116-127. 
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The children at Grant Park were divided into two groups^ one of which 
actually worked with the BR! program (N=6)* while the other was exposed to 
behavioral techniques designed specifically to teach the stated objectives 
of the BRL program (N=3), The scores for these two groups were combined 
in comparing Grant Park to the BRL alone group* Thus* the penultimate 
sentence in the preceding paragraph is not strictly correct in attributing 
the significantly greater increase at Grant Park to the BRL with behavior 
modification* 

Xt may have been the case that the Grant Park group that used behavior 
modification alone was superior to the group that used the BRL in conjunction 
with behavioral techniques. The small number of students in the two groups 
at Grant Park do not permit the resolution of this point; however., since 
the Laboratory for Applied Behavior Research will conduct a full scale 
study during the 1971—72 school year in the Atlanta Public Schools it should 
be possible to answer the question at a later date. 

In addition to the comparison between the Grant Park ;.hlt4ren and those 
in the BRL alone group, both of these groups were compared to the control 
group (N— 15) at Williams. The Mann— W hi tncy U test was used in making these 
comparisons. Both the Grant Park group and the BRL alone group showed 
significantly greater increases from pretest to posttest than did the control 
group at Williams. The U value for the comparison of the Grant Park group 
to the control group was 25 ^ a value that is significant at the „Gl level. 

The comparison of the BRX alone group to the Williams group yielded a U value 
of 144.5. This value is significant at the .001 level. (Again., to eliminate 
the suggestion that these groups may have been initially different from one 
another., the Mann-Whitney U test was used to discover whether the pretest 
scores for the three groups differed significantly from one another. The 
results of using this test showed that the three groups did not have 
significantly different pretest scores). 

The BRL— Sullivan Program in Prekindergarten 

The BRL program used in the Pre-K classes during the summer of 1971 
am eared to be a useful tool in teaching basic pre-reading skills to the 




children with whom it was used. The subjective impression of the evaluator 
was that the BRL program was easy to use and held the children f s attention 
for reasonable periods of time* Additionally ^ the pretest and posttest 
data showed that those children exposed to the BRL program did significantly 
better when tested than did those children who were not exposed to the 
program. 

In addition to the evaluation of the BRL program conducted by the 
Laboratory for Applied Behavior Research,, the teachers who used the BRL 
program were asked to respond to a questionnaire about the program * the 
results of this questionnaire are shown in Table 2, 

The teachers who used the BRL program were also asked to write 
recommendations based upon their experiences during the six week summer 
program. These recommendations are reproduced exactly as they were 
written: 



Recommendations for BR L 

1* We recommend that the Sullivan Kit be continued In the kindergarten 
and prekindergarten this fall^ and be used next summer and the 
following fall because: 

a. Directions are very good and are sequential. 

b. Teachers and children enjoyed it. 

c. Lessons correlate with other subjects in teaching the 
basic skills. 

2. We recommend that each school that use the BRL also use the enrich- 
ment kit . 

3. We recommend that all kindergartens use some type of formal reading 
program, 

4. Make !I Red !! red the color in the book is confusing with the color 
orange, 

5. Let the testing people also work with the children* 
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THE PEABODY REBUS 



Description of the Rebus Program 

Xn 19&5 a Rebus Reading Series was developed by Richard W. Woodcock 
as an experimental program for teaching reading to mentally retarded 
children. Out of this experimental series, the Peabody Rebus Reading Pro- 
gram (REBUS) was developed. 

The present REBUS has been tested with a variety of subjects inclu- 
ding kindergarten, Head Start, 1st grade, mentally retarded children, 
emotionally disturbed children, and with remedial readers. According to 
its authors/ the REBUS is primarily intended for use in kindergarten 
and 1st grade. 

The rationale behind the REBUS is to first teach the child a vocab- 
ulary of symbols which are easier to learn than spelled words. After the 
basic reading skills have been acquired, spelled words are slowly faded in 
(i.e., substituted for the symbol) . Xn behavioral terms, the REBUS attempts 
to reinforce the child for reading before he has acquired written language, 
thereby making the process of reading acquisition more interesting to the 
child , 

The REBUS consists of 5 books; 3 programmed workbooks and 2 readers. 
Workbook 1 presents a vocabulary of 35 symbols. Short sentences are 
formed from these symbols. Workbook 2 extends the vocabulary to 68 symbols. 
It also introduces certain skills involving several consonant and vowel- 
consonant • combinations , Workbook 3 extends comprehension skills and 
introduced phonic skills. Reader I extends the vocabulary to 120 symbols, 

40 of which are transitioned into spelled words. Reader II extends the 
reading vocabulary to 172 symbols, 122 of which are spelled words. 

The pre— kindergarten summer program was concerned mainly with Book X. 
Book I is divided into a series of frames, four to a page. There are two 
basic types of frames: those which require teacher involvement (frames 



^Richard W, Woodcock, Charlotte R. Clark, and Corn© 
"Its Fun to Read With REBUS," American Guidance Services 



Clark, and. Cornelia Oakes Davies. 
Guidance Services, Circle Pines, p. 6. 
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usually concerned with new symbols new skills ^ or reviews) and those 
which are worked individually by the student. Frames are grouped into 
sections which begin with several "teacher involvement" frames^ then 
proceed to several individually worked frames which end with a "Stop" 

sign. 

The "Stop" sign signals the end of a section— the pupil then lets the 
teacher know he has finished,, she checks his work and if she feels the 
pupil is doing well, she moves on to the next section of teacher involve- 
ment frames. 

The pupils* responses are made by dipping the eraser end of a pencil into 

a "water-well" (a moderately wet sponge) and then wiping the eraser end across a 
lined response area beneath the chosen answer. If the child has chosen 
the correct answer, the lined area will turn greenj if he has chosen the 
wrong answer, it will turn red. 

The Rebus Program With Behavior Modification 

Behavioral techniques were introduced in one class where the Peabody— 

Rebus Reading Program was to be used. This prekindergarten class was at 
Edward 5. Cook Elementary School and had an enrollment of 40 students * 36 
of which were in class for more than 15 days. A total of 15 children were 
exposed to the Rebus at some time during the 6 week session. The teacher 
did not feel that the remaining students exhibited the necessary social, as 
well as academic, skills to perform on the REBUS. 

The strategy in this class was to allow the teacher, Mrs. Sally Jackson^ 
to run the program her own ways and the consultant would implement techniques 
of behavior modification where he saw fit. The teacher was in complete agreement 
with this strategy. Following is a chronological description of the introduction 
of behavior techniques in the classroom. Refer to Table 3 for a diagram of 
these int ervent ion. 

On June 28 , Group I (5 children chosen by Mrs. Jackson as being the most 
mature) was introduced to Book One. The teacher asked the children to "read" 
(collectively) the symbols in frame .1. The children were also allowed to 
look through the book. 
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On the following day the children were again asked to "read" symbols 
in frame 1. The consultant suggested that this be done individually in 
order to make certain that all 5 children could read the symbols. The 
children were then asked to point to the large symbol in each subsequent 
frame in this section and to find the answer with their finger* Some 
pupils needed help with this., as they did not seem to understand the 
concept of matching. Pencils and "water-wells" were then passed out* 

The teacher explained how the items were to be answered and that "green 
means correct" and "red means incorrect". Mrs. Jackson went through 
each frame with the children and had them first point to the correct 
answer tfith their finger and after making sure each child had the right 
answer * the pupils were allowed to mark the correct answer. The above 
procedures were quite tedious because 1) some children seemed to have 
difficulty understanding the concept of matching* 2) it was difficult to 
keep the children attending to the taskj 3) some children seemed to lack 
enough confidence to make a decision by themselves; and 4) quite often 
a child would mark both answers just to see the colors appear. 

Another group of 5 pupils (Group II) was also begun on this day. 

These children were introduced to the book and were asked to read 
(both collectively and individually) the symbols in frame 1 . 

On June 30., Group II was seen first and they worked through the first 
section (frame 1 =* 8 ) in the same manner as Group I, However, this time 
it was suggested that the teacher explain that "green is good" and "red is 
bad" since these children had probably had more experience with the 
terms "good" and "bad" than with "correct" and "incorrect". 

After helping the students with frames 9 and 10 (teacher involvement 
frames) the childrei^were allowed to work on the second section (frames 
11-19) by themselves. The teacher helped only in turning pages and in 
putting a child on the correct frame. The percent of items correct on 
this section for these pupils was: 88^*11^* 0% ? and Q%* 

Group 1 was brought together later in the day for work on the REBUS 
Program. Since Group II had not done well on individually worked frames * 
the teacher helped each child in Group I find the correct answer for frames 
11-19. Subsequently* all children performed at a 100% correct level. 
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The suggestion was then made to Mrs. Jackson that "Front Loops" 

(a sugar-coated fruit flavored cereal) be given to those student who made 
no errors in a particular section. It was also suggested that she shape 
her students T behavior by giving "Froot Loops" for correct answers during 
the "teacher involvement" frames. The purpose of this intervention was ; 

1) reduce the reinforcement in marking a wrong answer simply to see the 
"magic" colors appear and 2) motivate the children to attend to the 
task and seek the right answer before marking. 

On the following day, July 1, only Group II worked on the REBUS 
program, (The teacher had many other activities planned and there was 
no time for Group I to work on REBUS). 

Mrs. Jackson explained to Group II that they could earn "Froot-Loops" 
by getting only green in their workbooks. She then helped the children 
with the "teacher involvement" frames before letting them work alone on 
the "individually worked" frames. However, the teacher failed to imple- 
ment the suggestion of shaping the children T s behavior by reinforcing them 
(with Froot-Loops") when they answered correctly on "teacher involvement" 
frames. The consultant did not remind her of this at this time. 

After going through the "teacher involvement" frames * Mrs, Jackson 
asked the students to go through frames 25-50 (individually worked frames) 
without pencils and point to the correct answers. Help was given to those 
wwho needed it. Pencils and water-wells were then handed out and,' after 
reminding them about the "Froof-Loops", the children were instructed to 
work by themselves until reaching the Stop sign. Two students were still 
marking both answers in each frame. After a few frames, the teacher 
gave them extra help for the rest of the section, (It was later decided 
that these two children should be dropped from the REBUS program in order 
to give them special training which would enable them to do the REBUS, 

In order to make it easier on the teacher, a child from Group I was moved 
to Group II, making 4 children in each group.) The 2 children who worked 
individually performed at a 80 % and 96 % correct level. 
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On the following day, the teacher instated the reinforcement program 
( "Froot-Loops") with Group I in much the same manner as with Group IT, 
however, this time she reinforced correct answers on the "teacher involve- 
ment" frames with "Froot-Loops" , These children were also asked to go 
through frames 25—50 without pencils at first. Only one child had 
trouble with this and he was given help. The other children worked indi- 
vidually withtheir dampened pencils and performed at the following levels: 
lQOf?, 96 %, and 92 % correct. 

On the following day, (July 5) the suggestion was made that students 
who made no errors would ccntinue to receive several Froot-Loops, but those 
missed only one would get 1 Froot-Loop. Those who missed more than one 
would not get any Froot-Loops. This suggestion was made after observing 
that when a child missed one item, his attention toward the task was greatly 
reduced, probably because the child realized he could not get any Froot- 
Loops. It was felt that differential reinforcement might remedy this. 



On the first day that the above changes were made, the teacher did 
not carry out the instructions in a smooth manner and the children seemed 
confused. Another factor which added to the confusion was the fact that 
a new skill was required of the students in this section of the book: 
not only did they have to match objects, but they had to match color 
and object. For example, the stimulus item may be a "blue box." 

The responses to chose from are: "yellow box," "blue box," and "blue 
bird." The child has to consider both color and object in oi^der to 
arrive at the correct answer. The students in both groups performed 
poorly on this section: 95 % (with help), 79 % (with help), 79 %o, 79 %, 

79 %, and 68 %. (Two children were absent and performed on this section 
at a later date under a token reinforcement procedure. These 2 children 
made 89 % and 79 % correct.) 



On the next school day the following instructions were given to Mrs. 
Jackson on a 5 x 8 card in order to help her run through the lessons more 




smoothly: 

1. Go through items having yellow circles with children. 

a. "Good, good job" if child gets green (+Froot-Loop) 

b. "Not good, wrong..." if child gets red (no Froot-Loops) 

2. Mark time on one book when children begin "individually" worked 
item 

3. Mark time in each book when child reaches "Stop" sign and raises 

4. Look over child's work 

a. all correct=5 Froot-Loops 

b. all but one correct=X Tf*oot« Loop 

c. More than 1 missed = no Froot Loops 



hand. 



d. More than 3 missed = send child to another section of room 
with aide for review of material (a check mark should be 
made under asterisk column of Progress Report) . 

e. Upon completion of daily Rebus program, children who have 
done good work should be praised and allowed to leave room. 
Children who have not performed well (missed more than 3) 
should remain in room for further review. 



Another change was also made. Cover sheets (heavy typing paper) 
were attached to the right hand page of the next 3 sections with small 
amounts of glue. This was done in order to help the child begin on the 
left page. When a child completed the left page, he could then tear off 
the cover sheet and begin working on the right page, (This procedure worked 
well, but the teacher still had to help some students begin at the top and 
work down) . 



The above procedures were used for the next 7 school days. 



Some 



days were used as review days, 
was as follows : 

FRAMES 103-115 



The percent correct over these seven days 



GROUP I 



GROUP IX 



S 1 

s 2 

s 3 

s 4 

S 1 

s 2 

s 3 

s 4 



79 % 

79 % 

68 

89 (with help) 

79 

74 

79 



FRAMES 120-139 

100 % 

100^> (with help) 

absent# 

100 (with help) 



92 

85 

92 



FRAMES 145-178 

95 % 

100 % 

absent# 

100 (with help) 

95 

95 

absent# 

100 



#these sections were done later under a token reinforcement procedure. 



A new group (six students) was begun during these seven days. However, 
the group was disbanded because it was felt that they lacked the prerequisite 
skills which would enable them to perform on the REBUS material. 

It was decided that these six children, along with the 2 children who 
were taken out of Group II, be put in a special group where these prerequisite 
skills could be taught. A behavior modification project using a multiple 
baseline across behaviors was designed to teach the following skills: 




^Donald M. Baer, Montrose M. Wolf, and Todd R. Risley. "Some Current 
Dimensions of Applied Behavior Analysis," Journal of Applied Beh avior Analysis, 
1968, Vol. 1, No. 1, p. 91. 
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1) distinguish red from green^ over from under* small from large* 
and top from bottom. 

2) understand the meaning of first* second* third. 

At this time a volunteer with both teaching experience and experience 
in behavioral engineering was sent to Cook by Dr. T. Ay 1 Ion. Mrs, Jackson 
decided to let the volunteer and the consultant conduct the REBUS program 
since the teacher was more interested in doing other things. The consul- 
tant agreed to this even though it meant giving up the behavior modification 
project with the special group. The AATS interns were asked to teach the 
above mentioned skills to this group in any manner they chose. 

On July l6 a token reinforcement program was begun with the REBUS 
groups. Green tickets were given for each frame in which a child got only 
a green "answer 11 . These tickets could be exchanged* as soon as the child 
reached the STOP sign* for candy and trinkets. It was also decided at 
this time that students should no longer get help on "individually worked" 
frames because extra help was becoming too reinforcing. Since the children 
were allowed to do poorly on some days and were not given help* a comparison 
of percent correct before and after the advent of token reinforcement is 
not practical. However* subjective observations made by the consultant 
show that the students attended to the task and seemed to be more motivated 
to perform well after tokens were in use than before. The token reinforcement 
program was used for the remainder of the summer program. 

After about three weeks in the special group* the AATS interns felt 
that 4 of their students were ready to begin REBUS. Therefore* a new group 
was started. After one session with this group* it was decided that one 
member was yet not ready for REBUS* since he did not seem to understand 
the concept of matching. He was sent back to the special group. 

Token reinforcement was used with the other three children and they 
performed at a fairly high percent correct level; 



sec 1 



sec 2 



88 % 



sec 3 

not done 



(44)* 

89 ^ 



(100% with help ^100%) 



6 5 % 



not done 



sections were done previously when Group III was first begun. 
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As with the BRL program* a Between group analysis will be discussed 
later concerning the REBUS with and without behavior modification. Fol- 
lowing is a description of the REBUS classrooms where behavior modification 
was not used. 

The Peabody— Rebus without Behavior Modification 

The Rebus Reading program was used not only at Cook but also in the 
Harris* Pitts * and Ware Pre-K classes. As was the case with the BRL pro- 
gram* the teachers at these latter schools were not given any specific 
instructions about using the Rebus program in their individual classrooms. 

The use of the Rebus program without behavior modification at Harris* 
Pitts* and Ware was evaluated in the same fashion as was the use of the 
BRL program without behavior modification at Jones* Luckie* and Slater: 
an evaluator paid randomly scheduled* unannounced visits to each school. 
Approximately three hours were spent at each school during each evaluation 
visit, 

Harris* The Pre-K program at Harris was directed by Miss Keller* 

She was assisted by both aides and volunteers in conducting classroom 
activities* Thirty-four children were enrolled in the Harris Pre-K class. 

Of these students* thirty-three attended for more than fifteen days* 

The first evaluation visit was made on Tuesday* June 29* 1971. 

Miss Keller had decided to conduct the Rebus program on Mondays* Wednesdays* 
and Fridays* thus no actual observation of the program* s use was made during 
this visit. Miss Keller had selected eight children for initial participation 
in the Rebus program. The criterion for her selection appeared to be her 
subjective evaluation of the child ! s ability to work independently* 

The second evaluation visit was made on July 16* 1971* Fourteen 
children were present at the time of this visit. The children using the 
Rebus program were placed in a separate room away from any distractions. 

Two groups of children using the Rebus were observed* The first of these 
groups (N— 2) was under the supervision of an aide. She initially used the 
supplementary Rebus cards to teach new symbols to the children. The chil- 
dren then received a brief review from the aide about marking the correct 
answer in each frame. Once the children were working in their books the 
aide gave individual help where it was needed* Most of her activity was 
directed toward pointing out missed frames and skipped pages. The children 
^ spent about fifteen minutes working in their books. 



The second group of children using the Rebus (N=3) was under the 
supervision of Hiss Keller, Sh:; started by having the children call out 
various symbols on the Rebus earns. Unfortunately^ in correcting the 
children's response. Miss Kelle- consistently ( and incorrectly) referred 
to the symbol representing "black ; as the "black ball". This led the chil- 
dren to confuse color symbols wit* the ball symbol. The children in the 
second Rebus group spent approximately thirty-five minutes working on the 
pro gram . 

In both the two groups the children stopped working whenever the 
instructor was helping another child. This slowed down each child's 
working pace and effectively precluded the use of a rate measure for 
indexing the children's progress with the Rebus. 

Pitts. The Pre-K class at this school was supervised by Miss Kahn. 

Two aides always assisted her in conducting class activities. Twenty- 
nine children were enrolled in the Pre-K class and twenty-two of them 
attended school for more than fifteen days. 

The first evaluation visit to Pitts was made on July 2, 1971- Six- 
teen children were present. Due to involvement with a special activity 
(making apple sauce) the Rebus was not conducted. Miss Kalin reported 
that the Rebus was normally used everyday except on those days when field 
trips were taken. She said that seven children were currently using 
the Rebus materials and she planned to begin use of the program with other 
children on July 6, 1971. She commented on the fact that the children 
seemed to have difficulty with the concept of a frame and that there was 
a tendency to confuse the Rebus symbols with one another. To facilitate 
learning the skills necessary for using the Rebus, Miss Kahn and reproduced 
pages from the Rebus book on Flip charts. These large charts were then 
used to teach the children how to mark answers, recognize frames, etc. 

The second evaluation visit was made on July 13, 1971* Twenty— two 
children were present. Five children were observed while they used the 
Rebus program. Of these five children, two (a six year old girl and a 
girl who had attended Pre-K previously) could work independently. The 
other children needed continuous supervision in working with their books. 
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Miss ICahn spent about ten minutes working with the children as a group. 

She used the Rebus cards to teach new words, review old words, build 
simple sentences for reading, and build !T f±ll in the blank 11 sentences 
for the children to work in their individual books. 

Three major facts were noted by the evaluator during the second visit 
to Pitts. The first of these was that the children had great difficulty 
in reading sentences which had color symbols in them. This may have been 
due to the fact that the color symbols in the Rebus program were so easily 
recognised and verbalised that they interfered with the recognition of 
less meaningful symbols. Second, there was a tendency on the part of 
the children to always start on the right hand page of their workbooks. 

They could not learn to start on the left hand page and usually had to 
be instructed by Miss Kahn to do S o. Third, Miss Kahn had instituted 
gross reinforcement procedures for rewarding children who successfully 
completed a Rebus section. She gave each child who completed his work 
a marshmallow. The children seemed to expect the marshmallow no matter 
how long they took to do a section in their books. The children spent 
an average of twenty minutes working with the program, although one boy 
was still working on his book when a fire drill necessitated his stopping. 

The third evaluation visit was made on July 21, 1971. Fifteen chil- 
dren were present. One of the two children who could work without super- 
vision had been absent and had just returned to class* Miss Kahn was 
forced to work with this girl alone, teaching her new words that the other 
children had learned already. This reduced the amount of time that could 
be spent with those children who were not so far advanced in the Rebus 
book. All told, five children worked in their Rebus books. The average 
time spent was about ten minutes * 

Ware , Miss Schwarts directed the Pre-K class at Ware, She was assisted 
by Aides, volunteers, AATS Interns, and NYC workers. Thirty children were 
enrolled in the Pre— K program. Of these children, twenty seven attended 
class more than fifteen days. 
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The first evaluation visit to Ware was made on June 30, 1971- Miss 
Schwarts had selected nine children to participate in the Rebus program. 
Eight of these children were present and were observed using the Rebus. 

The children were seated on the floor in a circle. Miss Schwarts asked 
the children, as a groups to name several of the large Rebus pictures. 

The individual children then named the small Rebus cards and matched them 
with the pictures .(Not all the children participated in this activity.) 

Next, Miss Schwartz, held a brief review of how to mark the answers in the 
Rebus book since the children did not seem to understand what action they 
were to take if they chose an incorrect answer. Miss Schwartz then had 
them work, under her supervision, on their own. The children spent about 
ten minutes in their books. 

The second evaluation visit was made on July 9* 1971* No Rebus 
activity was conducted on that date due to the visit of the school doctor. 

The children worked under the close supervision of Miss Schwarts, who turned 
pages for them, pointed out skipped frames, and asked them leading questions 
about their answers. She also kept up a running stream of verbal reinforce- 
ment for their performances. The five children read their Rebus books aloud . 
This generated a great deal of confusion and, coupled with Miss Schwartz ! s 
comments, produced a noise level altogether too high for effective work. 

In addition to the children working with Miss Schwartz, two other children 
received individual instruction from an aide and an NYC worker. One of 
these children simply worked in her book under continuous supervision, 
while the other was given a review of previous materials. The average 
total time spent on the Rebus by all the children in both groups was 
fifteen minutes . 
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The Results of Using the Peabody-Rebus 
Program in Pre kinder gar ten Classes 

Table 4 gives the percent of students at each school who were 

exposed to each section of the REBUS, Book X. A total of 54 children in 

the Pre-Kindergarten Summer Program were exposed to the REBUS, (42% of all 
the children enrolled in schools where the REBUS was used) . Out of these 
54 students, 40 &74%( finished at least l/3 of Book Xj 13 (24%) finished 

at least 2/3 of Book X| and 3 (6%) finished Book X, 

A continuous measure of progress is possible with the REBUS because 
it is individualised and because the children progress through the book by 
11 answering” frames. For this reason, a pre-post test was not necessary 
in comparing the Rebus with behavior modification to the REBUS without 
behavior modification. 

Teachers using the REBUS program were asked to record rate data (i,e. 
number of frames completed per minute) on their students by timing them 
on each section of ,T individually worked” frames. From this data, rate 
correct and rate incorrect can be computed. This type of data is a more 
accurate description of behavior than is percent data, i.e. a child who 
marks 5 correct frames per minute is performing at a different level than 
one who marks 1 correct per minute even though the percent correct may 
be the same for both students. Also, certain analyses are possible with 
rate data that are not possible with percent data. For example, when 
both rate correct and rate incorrect are high, the child is working too 
rapidly* 

Periodic observation at the schools showed that rate data were in 
many cases invalid because the teacher would often interrupt the students, 
thus affecting their rates. It was also noticed that when the teacher 
was correcting the work of one student, the other students would stop 
working and watch. It was therefore decided that percent correct data 
(not rate data) should be used in any quantitative analysis of performance 
on the REBUS. 
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To further aid in the analysis of the REBUS, the teachers were asked 
to mark, on progress reports provided, those sections in the REBUS where 
the child received help during "individually worked” frames. This was done 
in order that these data could he omitted when comparing the performance 
of children at one school with those of another. However^ there is reason 
to believe that teachers in fact did not always report the times when help 
was given. At Cook, children received help on 30 occasions out of 108. 

At Ware, only 5 sections out of 116 were marked as "help given". At 
Harris, 19 sections out of 77 were marked as "help given". Help was never 
given at Pitts, according to the Progress Report. In addition to these 
discrepancies , incongruities were found between observations made by an 
evaluator and the Progress Reports filed by the teachers. 

An analysis of the percent correct data excluding the Ilhelp given" 
sections shows that the scores for children at Cook (where behavior modifi- 
cation was used) were significantly higher than those at Harris (t=2.79l7j 
with 18 df; p <\0i, one tailed)# but do not differ from those at the 
other three schools combined. 

Because of the discrepancy in the reporting of help given on certain 
sections, it was decided that an analysis should be done on the percent 
correct scores including the help given scores. When these data were 
included, the percent correct scores at Cook (Rebus with behavior modi-* 
fication were significantly higher than those at the other three schools 
combined (t— 2,7310, with 52 dfj p<^ .01, one tailed) where only the Rebus 
material was used. 

There are two factors which could account for these results: l) the 

help given at Cook could have influenced the scores more than the help 
given at the other schools, 2) the behavior modification techniques used 
at Cook had the effect of producing higher scores. The fact that per- 
cent correct scores excluding "help given" scores are significantly 
higher at Cook than at Harris (the only school where the percent of 
"help given" information was near that of Cook) lends strength to the 
second hypothesis. 




The Rebus in Prekindergarten 



Tills section will deal with the usefulness of the REBUS in a pre-kinder- 
garten Summer Program. It should be kept in mind that the comments made 
in this section refer to the population of 4 year old children seen in 
this summer’s program. We do not wish to generalize* in the absence of 
data^ to all 4 year old children. 

An important consideration in evaluating a programmed instructional 
material* such as the REBUS * is error rate. A low frequency of errors is 
necessary if a behavior* in this case reading* is to be acquired. Acquisition 
occurs when the behavior is reinforced by correct answers. 

The average percent incorrect across students in all REBUS classes 
was 21^3. Only 4 children exhibited a frequency of errors below 10^. 

Eight children exhibited a frequency of errors above 30^. It is felt 
that this frequency of errors is too great for the REBUS program to be 
useful . 

The reason for the high frequency of errors probably lies in the 
prerequisite skills which are necessary for performance on the REBUS. 

Some of these skills* as mentioned before * are: matching* working left 
to right and top to bottom* understanding the concepts of first* over* 
under. Attention span also seems to be a problem associated with this 
population of children. 

The teachers using the REBUS program were asked to answer a questionnaire 
at a post— service meeting held on August 3* 1971* which dealt with their 
subjective evaluation of the REBUS program. The results may be found 
in Table $* 

The REBUS teachers were also asked to meet and prepare a list of 

recommendations concerning the REBUS program. These recommendations are 

reproduced here without comment. 

Children need more readiness and time to develop certain basic 
skills and understandings before they can successfully use work- 
books . 

Instructions and terms aren’t clear to children. 

More practice sheets are needed. 

Lessons are too long for most children T s attention spans. 
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5* The program takes a great deal of the teachers time. 

6. It takes too long before children can work independently^ 
follow order of frames, go on from page to page without teachers 
help. 

7. This material doesn T t seem particularly suited for deprived or 
disadvantaged children. 

8. All that we have read and been taught has stressed the importanc 
of pre-school children learning in an unstructured^ informal 
situation* We feel that this program contradicts our beliefs 
concerning the instruction of pre— school children. 
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Summary and (Conclusions 



It is not possible to compare the BRL and Rebus programs with 
one another in terms of a set of objective criteria that could be equally 
applied to both programs. The BRL program is essentially a group oriented 
program that requires the teacher to be present whenever it is used. It 
provides no means for continually assessing a child T s progress. The Rebus, 
on the other hand, is used individually by each student and theoretically 
can be used without the teacher’s direct supervision. The Rebus also 
provides a means of continually recording an individual child’s performance. 

If one were forced to choose between the two programs for use with 
a Pre-Kindergarten class similar to those encountered this summer, the 
program chosen should be the BRL. The BRL program does not demand that 
the children using it have complex behaviors (such as frame marking and 
matching) already In their repetoire. Additionally, the BRL program in 
the Pre-K classes can be used repeatedly. With a!3 children, while the 
REBUS cannot. 

It should be pointed out that there is a potential alternative to 
either the REBUS or the BRL programs. This alternative consists of 
using behavioral techniques to teach specific skills and concepts 
necessary for reading. Although the Summer Pre-K project was not suffi- 
ciently large to permit a quantitative investigation of this method, the 
data obtained seemed to indicate that behavioral techniques were effec- 
tive in teaching pre-reading skills. Additional evidence for the 
effectiveness of behavioral techniques may be seen from the fact that 
some of the Pre-K teachers spontaneously used operant principles in 
their classes, even though they had no contact with the consultants 
from the Laboratory for Applied Behavior Research. Some of the tech- 
niques developed by these teachers are listed below- 



1. Reinforcement of appropriate behavior 



, — at Ware balloons and candy were made contingent upon performance of 
specific activities (language motto , painting, puzsels, REBUS, etc.) 
and good verbal description of those activities to the teacher, 

—Marshmallows were made contingent upon completion of a set of "REBUS" 
frames at Pitts, 

2, Use of discriminative stimuli 

—Various piano notes signaled the beginning of specific activities at 
Luckie , 

A bell was used at Williams as a signal for assembly and clean-up. 

3, Use of competing responses to prevent unwanted behavior 

Having children tiptoe from classroom to another building where a film 

was to be shown precluded running and to a certain extent talking at 

Harris, 

4 , Satiation 

— At Harris, demonstrating the projector and allowing the children to 
touch the film prior to the movie caused the children to pay more attention 
to content of movie and less to the workings of the projector, 

5, Shaping 

When the doctor came to Ware, the teacher reduced anxiety by having 

the children use the stethoscope on one another and having them "check" 
each others fingers in a simulation of the hematocrit test, 

6, Verbal modeling of behavior 

—When children at Pitts were to make an apple out of paper, the 
teacher described the process in detail and then had the children repeat 
the instructions as a group using a "fill— in— the— blank" technique, e,g, 

"cut the apple out of the paper, (red)," 



All of these procedures are adaptable to other primary classrooms and can 
be carried out successfully by paraprofessionals. 
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APPENDIX 



TABLE 1 

Within School Change From Pretest To Posttest 
For Those Schools Associated With The BRL Program 



School 


N* 


Direction 

of 

Change 


T 


P 


Grant Park 


9 


Increase 


3.0 


^.01 


Jones , J. M, 


18 


Increase 


2,5 


<-.005 


Luckie 


12 


Increase 


6.0 


.005 


Slater 


18 


Increase 


0.0 


L .005 


Williajns 


34 


Increase 


3.8 





The Wileoxon Test does not use individual scores of zero. Those 
students at each school who made the same score on the posttest 
as they did on the pretest were dropped from the analysis. 



-K-;! Not Significant. 
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TABLE 2 



Results of the Teacher Questionnaire 
Dealing with the BRL Program 



Agree 


Not Sure 


Disagree 


3 




i 


4 






4 






I 




3 


4 






4 






1 




3 


2 


i 


1 




i 


3 


2 




2 



1. The reading program used in my classroom was 
applicable to 4 year olds. 

2. The re ading program used in my classroom r equired 

much of the teachers time. < 

3. I feel that the children enjoyed working on the 
reading program, 

4. If I had the choice, I would not use the reading 
program again in a six week program, 

3. The particular reading program used in my class 
should be continued with the same children this 
fall when they begin kindergarten. 

6, The material presented in the reading program was 
useful in preparing the children for kindergarten. 

7. The reading program was more important than anything 
else I did in preparing the children for kindergarten.! 

8. I could have taught the same material that is j 

presented in the reading program without having to 
use the program. I 

9, I could have done a better job of teaching the materic. 
presented in the program without using the program. : 

10. Absences from class were a great inconvenience as j 
far as the reading program is concerned. { 
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TABLE 3 



Reinforcements 



Bate 


Type of 
Reinf or c ement 


Status of Groups 


6/28 


None 


Group X begun 


6/29 


None 


Group II begun 


6/30 


None 




7/01 


Froot-Loops 


2 Ss dropped from 
Group II 


7/02 


Froot— Loops 




7/06 


Froot— Loops 




7/07 


Froot— Loops 




7/08 


Froot-Loops 




7/09 


Froot— Loops 




7/12 


Froot-Loops 


Group III begun 


7/13 


Froot— Loops 


Group III disbanded 


7/U 


Froot— Loops 


Special Group begun 


7/15 


Froot«Loops 




7/16 


Tokens 




7/19 


Tokens 




7/20 


Tokens 




7/21 


Tokens 




7/22 


Tokens 


Group III restarted 


7/23 


Tokens 




7/26 


Tokens 




7/27 


Tokens 





Other 

Information 



Green = Good 
Red = Bad 



3x8 Card and 
Cover Sheet 



Volunteer given 
responsibility of 
REBUS - 



Last day with REBUS 
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TABLE 4 





Per 


Cent of Students 
Section of REBUS 


Exposed to Each 
Workbook I 






Section 


Per Cent 
of Students 
at Cook 


Per Cent 
of Students 
at Harris 


Per Cent 
of Students 
at Pitts 


Per Cent 
of Students 
at Ware 


i 


38 


30 


41 


50 


2 


35 


30 


41 


50 


3 


28 


30 


41 


50 


4 


23 


30 


38 


47 


5 


20 


27 


31 


43 


6 


20 


16 


14 


43 


7 


20 


14 


7 


40 


8 


20 


14 


7 


33 


9 


18 


5 


7 


30 


10 


10 


5 


7 


17 


11 


10 


5 


7 


3 


12 


8 


0 


0 


3 


13 


8 


0 


0 


3 


14 


8 


0 


0 


3 


15 


5 


0 


0 


3 
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TABLE 5 



Results of the Teacher Questionnaire 
Dealing with the REBUS Program 



Agree Not Sure Disagree 



1 


2 


1 


1 . 


The reading program used in my classroom was 
applicable to 4 year olds. 


2 


i 


1 


2. 


The reading program used in my classroom required 
much of the teachers time. 


4 






3. 


1 feel that the children enjoyed working on the 
reading program. 


3 




i 


4- 


If I had the choice* I would not use the reading 
program again in a six week program* 


0 


2 


2 


5. 


The particular reading program used in my class 
should be continued with the same children this 
fall when they begin kindergarten. 


3 




1 


6. 


The material presented in the reading program was 
useful in preparing the children for kindergarten. 






4 


7, 


The reading program was more important than anything 
else I did in preparing the children for kindergarten. 


2 


1 


1 


8, 


I could have taught the same material that is 
presented in the reading program without having to 
use the program. 


1 


2 


1 


9. 


I could have done a better job of teaching the 
material presented in the program without using 
the program. 


1 




3 


10. 


Absenses from class were a great inconvenience as far 
as the reading program is concerned. 
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TEST INSTRUMENT USED WITH THE BRL PROGRAM 





SULLIVAN SKILLS AND CONCEPT SURVEY 
( : CORRECT RESPONSE) 


1 


SCHOOL 




fFWRT 1 


TEST 2 




CHILD 


DATE 2 


DATE : 


AGE 

CARD ONE 






1. 


Point to the top of the card. 


i 








2, 


Point to the bottom of the card* 


2 








3, 


Show me the line that goes down the middle 












of the card* 


3 








It. 


Show me your right hand. 


1) 








5* 


Show me your left hand. 


5 








6 . 


Point to the left side of the card. 


6 








7. 


Do you see more than one square on the left 












side of the card? (yes) 


7 








8. 


How many do you see? (2) 


8 








9, 


How many circles do you see on the right side 












of the card? (1*) 


a 








LO. 


Point to the first color in the top row* 


10 








LI. 


(POINT) What is the name of the first color 












in the top row? (green) 


11 








L2. 


(POINT) What is the name of next color? (orange) 


la 








L3. 


(POINT) What is the name of the next color? 












(black ) 


13 








Lit. 


(■POINT) What is the name of the next color? 












(red) 


lU 








L5. 


How ? point to the first color in the middle row* 


15 








LC, 


(POINT) What Is the name of the first color 












in the middle row? (pink) 


16 








L7. 


(POINT) What is the name of the next color? (white 


)17 








l8. 


(POINT) What is the name of the next color? (blue) xd 








LP. 


(POINT) What is the name of the next color? 












(brown or tan) 


IP 
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CARD TWO: ____ 

Look at the top of the card 

20. Put your finger on the top left square* 

21. (POINT TO TOP LEFT) What is the name of 
that animal? (tiger) 

22. (POINT) T Tiat is the name of the next animal? 
(snake) 



TEST 1 



TEST 



20 

21 

22 



23. (POINT) -What is the name of the next animal? 

(fox) 23 

2k . ( POINT TO BOTTOM LEFT) What is the name of 

that animal? (bird) 2k 

25* (POINT) iTiat is the name of the next animal? 

(lion) 25 

26 . (POINT) ITiat is the name of the next animal? 

(giraffe) 2 ^ 

27* Me have pictures and letters on this card. Point 

to some of the letters* (anv letters will do) 27 

28. What letter is under the tiger? (a) 88 

2 9* What letter is over the giraffe? (b) 29 

30 * What letter is over the bird? (a) 30 

31 . What letter is under the lion? (c) 31 

32. What letter is under the fox? (b) 32 

33. TT hat letter is over the lion? (c) 33 

34. What letter is under the giraffe? (d) 34 

35. What letter is under the snake? (c) 35 

3 6 . Can you read the letters in the red hox at the 

top of the card? Read them for me, (d, a, d) 38 

37. Those letters spell a word. Can you read the word? 

(DAD) " 37 
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